(1) Field of the Invention
The present invention relates to a motion picture reconstructing method and apparatus which provide a function for manipulation of digital visual data for an operator to manipulate an object or objects in a motion picture.
With rapid development of moving picture processing technology, functions of object manipulation and others of digital audio-visual data have been demanded, and these functions are to be supported by MPEG-4 (Moving Picture Experts Group Phase 4). The object manipulation will offer several capabilities to an operator, including the functions of object translation, object color change, object shape change, object replacement, and object deletion with respect to one of the independent objects of a motion picture.
MPEG-4 is an emerging coding standard that supports new ways for communication, access, and manipulation of digital audio-visual data. The object manipulation mentioned above is one of various functions to be supported by MPEG-4.
(2) Description of the Related Art
FIGS. 1A, 1B and 1C show the basic concept of reconstructing a motion picture in which an overlay is removed or deleted to view the concealed video beneath the overlay.
An original picture is shown in FIG. 1A. In the original picture, background objects including a shaded vehicle 2 and a shaded house 3 are overlaid by a white vehicle 1. Such an object as the vehicle 1 in the original picture will be called the overlay.
An intermediate picture in which the overlay is removed is shown in FIG. 1B. In the intermediate picture, the vehicle 1 is removed, and corresponding areas are left in white.
A final picture which is reconstructed from the original picture to view the video beneath the overlay is shown in FIG. 1C. In the final picture, the corresponding areas of the vehicle 2 and the house 3, concealed by the vehicle 1 in the original picture, are generated.
In order to carry out the reconstruction of the motion picture, it is necessary to estimate values of the concealed picture elements of the background objects when the graphics overlay is removed. The estimated picture elements are used for substitution for the corresponding picture elements of the final picture, so that the overlay is removed and the picture beneath the overlay looks natural in the reconstructed picture.
In a case of a motion picture in which the background objects are fixed or not moving, a conventional method which estimates values of the concealed picture elements within the motion picture of the present frame is known. In the conventional method, corresponding picture elements of the background objects of the frames preceding or following the present frame are searched, and the values of the concealed picture elements of the present frame are estimated from values of the picture elements of the background objects found in the preceding or following frames.
However, in a case of a motion picture in which the background objects are moving, it is impossible for the above conventional method to estimate the values of the concealed picture elements of the present frame from the values of the corresponding picture elements of the background objects of the preceding or following frames. In this case, the positions of the background objects in the motion picture are changed from frame to frame. The conventional method does not teach estimating or tracking the positions of the picture elements of the background objects of the preceding and following frames.
Further, in order to carry out the conventional method, it is necessary to store the data of the picture elements of the background objects of all the preceding and following frames including the present frame into a frame memory. Therefore, the conventional method requires the frame memory to have a large amount of storage capacity. In addition, the above conventional method is not suited to perform a real-time data processing for the reconstruction of a motion picture when the overlay is removed.